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EXECUTIVE  SUMMARY 

For  a  number  of  reasons,  technical  publications  for  military  systems  are  now 
delivered  as,  or  converted  to,  electronic  format.  These  reasons  include  limiting  space 
and  weight,  providing  timely  access  to  technical  information  and  timely  updates,  and 
avoiding  deterioration  common  to  paper  products.  However,  there  are  a  number  of 
problems  that  can  make  today's  electronic  documents  less  than  acceptable.  Limited 
access  to  computer  support,  complex  and  unintuitive  user  interfaces,  poor  presentation 
of  graphics  or  diagrams,  and  restricted  mobility  for  technicians  are  just  some  of  the 
issues  that  force  maintenance  personnel  to  revert  to  paper  versions  of  technical 
manuals.  In  order  to  discuss  the  domain  and  to  present  current  research  in  this  area,  a 
workshop  was  conducted  23  January  2002  at  the  NAVAIR  Orlando,  Training  Systems 
Division.  The  purpose  of  this  report  is  to  highlight  the  proceedings  of  the  workshop  and 
further  define  the  goals,  technologies,  and  issues  related  to  this  research  effort. 

The  following  is  a  list  of  presentations  given  during  the  workshop: 

Human  Factors  in  Naval  Aviation  Maintenance  Related  Mishaps:  Focus  on  Publications, 
CDR  John  Schmidt,  Naval  Safety  Center 

IETM  Development  and  Intelligent  Diagnostics,  Dr.  Douglas  Towne,  University  of 
Southern  California/Behavioral  Technology  Laboratory 

Advance  Search  and  Display:  Dynamic  SuperManuais  with  Latent  Semantic  Analysis, 

Dr.  Thomas  Landauer,  Knowledge  Analysis  Technologies 

Spoken  Language  Dialog  and  Improved,  Adaptive  Design  of  Advanced  IETM  Interfaces, 
Dr.  Alex  Rudnicky  and  Dr.  Jane  Siegel,  Carnegie  Mellon  University 

Maintenance  Support  Technologies,  Mr.  Mike  Kendall,  Task-Based  Technologies,  Inc. 

The  Use  oflETMs  for  Operations,  Doctrine,  and  Training,  TMCM  (EOD)  Mark  Nelson, 
COMEODGRU  TWO 

As  part  of  the  Capable  Manpower  Future  Naval  Capabilities  (FNC)  program,  the 
Office  of  Naval  Research  (ONR)  is  supporting  a  research  effort  addressing  the  needs  of 
the  technical  manual  community.  While  there  is  a  wide  range  of  potential  research 
issues  associated  with  technical  manuals,  the  focus  of  the  Intelligent  Performance 
Support  and  Training  effort  is  the  development  and  evaluation  of  various  technologies 
to  support  Interactive  Electronic  Technical  Manuals  (IETMs).  These  technologies 
include:  the  use  of  device  models  and  intelligent  tutors  to  exploit  the  use  of  computers 
in  aiding  and  training  technicians  through  their  IETM;  the  use  of  a  machine  learning 
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technology,  Latent  Semantic  Analysis,  to  assist  in  search  and  navigation  tasks  within  an 
IETM;  and  the  use  of  wearable  computers  and  spoken  natural  language  dialogs  to 
support  a  voice-activated,  hands-free  working  environment. 

The  first  section  of  this  report  describes  current  issues  related  to  the  use  of 
technical  manuals  in  general,  and  more  specifically,  problems  associated  with  the 
current  generation  of  IETMs.  The  next  three  sections  provide  descriptions  of  the 
research  areas  within  the  Intelligent  Performance  Support  and  Training  effort.  The  final 
section  of  this  report  describes  issues  raised  during  workshop  discussions. 
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INTRODUCTION 

As  part  of  the  Capable  Manpower  Future  Naval  Capabilities  (FNC)  program,  the 
Office  of  Naval  Research  (ONR)  is  supporting  a  research  effort  addressing  the  needs  of 
the  technical  manual  community.  While  there  is  a  wide  range  of  potential  research 
issues  associated  with  technical  manuals,  the  focus  of  the  Intelligent  Performance 
Support  and  Training  effort  is  the  development  and  evaluation  of  various  technologies 
to  support  Interactive  Electronic  Technical  Manuals  (IETMs).  A  workshop  was 
conducted  23  January  2002  at  NAVAIR  Orlando,  Training  Systems  Division,  to  discuss 
the  domain  and  to  present  current  research  in  this  area.  The  purpose  of  this  report  is 
to  document  the  proceedings  of  the  workshop  and  further  define  the  goals, 
technologies,  and  issues  related  to  this  research  effort. 

Recently,  technical  publications  for  military  systems  are  being  delivered  with,  or 
converted  to,  electronic  format.  Space  and  weight  issues  related  to  storage  and 
transfer  of  technical  manuals  are  the  primary  driving  forces  behind  conversion  to 
electronic  format.  Whether  the  technician  is  deployed  in  the  field,  working  on  the  flight 
line,  or  stationed  onboard  a  ship,  these  issues  are  critical.  This  is  exacerbated  by  the 
fact  that  many  present  day  systems  are  complex  and  interconnected  -  making 
maintenance  documentation  that  much  more  "bulky".  Quick  access  to  critical 
information,  without  the  need  for  thumbing  through  volumes  of  documentation,  is 
highly  appealing.  Furthermore,  updates  to  paper  documentation  due  to  system 
changes  or  revealed  errors  are  cumbersome  under  the  best  of  circumstances  and  all  too 
often  not  performed  or  unseen  in  others.  This  leaves  the  technician  operating  with  out¬ 
dated  or  erroneous  information  that  may,  in  some  cases,  prove  dangerous  to  the 
technician  or  operators.  Other  relevant  issues  include  the  fact  that  by  its  very  nature, 
paper  deteriorates  and  may  do  so  rapidly  in  harsh,  military  environments.  Electronic 
documentation  aids  in  the  updating  process  and  ensures  that  the  document  remains 
intact. 


There  are  two  types  of  electronic  technical  publications:  the  Electronic  Technical 
Manual  (ETM)  and  the  Interactive  Electronic  Technical  Manual  (IETM).  While  the  term 
ETM  can  encompass  a//forms  of  electronic  documentation,  it  is  typically  used  to  refer  to 
raster  scanned,  page  oriented  documents  or  Standard  Generalized  Markup  Language 
(SGML)  tagged  scrolling  documents  that  may  contain  internal  hyperlinks.  The  IETM  is  a 
more  sophisticated,  interactive  electronic  document  that  includes  links  to  external 
resources,  such  as  diagnostic  aids  or  training  materials,  and  can  be  hierarchically 
structured  to  capitalize  on  data  reuse.  IETMs  are  categorized  as  classes  one  through 
five.  A  more  specific  description  of  the  classifications  of  ETMs  and  IETMs  is  provided  in 
Table  1. 
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The  notion  of  electronic  documentation  is  not  new.  From  the  Navy's  perspective, 
the  push  for  electronic  documentation  began  in  the  1980's  with  the  notion  of  the 
"paperless  ship."  In  fact,  there  are  many  systems  that  are  and  have  been  supported 
by  electronic  documentation  for  some  time.  Further,  there  is  a  wide  range  of 
characteristics  that  an  electronic  document  may  possess  and  there  are  a  number  of 
vendors  of  IETM  authoring  and  viewing  products.  There  also  are  standards  and 
guidelines  available  that  aid  in  producing  electronic  documentation.  For  example,  MIL- 
PRF-87268A  describes  how  an  IETM  should  look  and  behave  for  the  reader  and  MIL- 
PRF-87269  describes  database  forms,  structure  and  key  IETM  controlling  mechanisms. 

PROBLEMS 

Given  the  need  for  electronic  documentation,  the  availability  of  tools  for 
development  and  delivery,  and  the  existing  guidance,  it  is  surprising  that  IETM 
applications  often  go  unused.  There  are  instances  where  basic  problems  interfere  with 
a  technician's  ability  to  use  an  IETM.  For  example,  an  IETM  may  not  be  used  on  the 
job  because  the  activity  lacks  computing  power  to  run  the  IETM.  Simply  stated,  it  is 
difficult  to  work  with  today's  technology  on  a  386  computer.  Flowever,  more  complex 
and  pervasive  issues,  such  as  the  poor  usability  of  the  documentation  or  computer 
interface  or  limited  skill  levels  of  technicians,  can  also  lead  to  under-utilization  of  what 
should  be  a  valuable  resource. 

Usability  Issues 

Maintenance  manuals  contribute  to  maintenance  error  if  they  contain  misleading 
information,  insufficient  information,  or  unclear  procedures  (Chaparro  &  Groff,  2001). 

In  IETMs,  the  navigational  environment  is  also  critical.  If  a  document's  interface  is  too 
complex  and  the  navigation  is  unintuitive,  users  have  difficulty  finding  and  following 
documented  direction.  Usability,  defined  as  "the  ease  with  which  a  user  can  learn  to 
operate,  prepare  inputs  for,  and  interpret  outputs  of  a  system  or  component"  (IEEE, 
1990),  will  impact  a  user's  ability  and  willingness  to  refer  to  the  document. 

Problems  with  maintenance  manuals  are  not  new.  User's  perceptions  of  the 
utility  of  a  manual  can  apply  equally  to  paper-based  documents  and  electronic 
documents.  In  a  study  conducted  for  the  Australian  Transportation  Safety  Bureau, 

*  67%  of  respondents  reported  having  been  misled  by  maintenance 
documentation, 

*  47%  reported  having  opted  to  perform  a  maintenance  procedure  in  a  way 
they  felt  was  superior  to  that  described  by  the  manual,  and 

*  73%  of  mechanics  surveyed  reported  failing  to  refer  to  maintenance 
documents  either  occasionally  or  often  (Flobbs  &  Williamson,  2000;  Chaparro 
&  Groff,  2001). 
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It  must  be  assumed  that  these  same  problems  are  also  inherent  in  electronic 
documentation.  Adding  to  that,  the  navigational  environment  within  the  IETM  can  also 
be  problematic.  If  an  electronic  document  is  too  complex  and  unintuitive,  the  user  will 
revert  to  older  paper  copies,  or  choose  to  ignore  prescribed  documentation  entirely. 
Therefore,  an  IETM  must  provide  information  that  is  not  only  correct,  it  must  allow  the 
user  to  access  that  information  easily  and  in  a  timely  manner. 

Unfortunately,  current  capabilities  predominantly  utilize  computers  as  a 
substitute  for  paper  rather  than  capitalizing  on  computing  resources.  Often,  the 
information  that  a  technician  needs  is  embedded  within  complex  diagrams  and 
schematics.  On  paper,  they  are  static  representations.  On  the  computer,  not  much 
changes  -  and  worse,  these  drawings  are  difficult  to  view  due  to  size  and  screen 
constraints. 

Finally,  the  working  environment  may  not  be  conducive  to  the  use  of  a  PC  or 
laptop.  Working  in  small,  confined  spaces  (e.g.,  underneath  a  large,  tracked  vehicle)  or 
in  environmentally  harsh  conditions  (e.g.,  the  engineering  spaces  onboard  a  ship  or  a 
loud  flight  line)  provides  little  room  and  poor  habitability  for  conventional  computers. 

Data  briefed  at  the  NAVAIR  workshop  indicate  that  in  the  last  decade, 
approximately  600  mishaps  in  U.S.  Naval  Aviation  had  a  maintenance  casual  factor.  Of 
those,  approximately  25%  were  at  least  partially  due  to  a  problem  with  publications 
(technical  manuals).  Considering  that  the  average  cost  of  each  mishap  is  approximately 
$1M,  there  is  a  considerable  cost  factor  involved.  The  Navy  Safety  Center,  with  support 
from  this  research  effort,  is  currently  refining  data  associated  with  publication  errors  in 
order  to  more  accurately  pinpoint  problem  areas  within  publications. 

Skilled  Workforce  Decline 


Further  complicating  the  issues  associated  with  IETM  usability  is  the  declining 
level  of  expertise  of  the  typical  maintenance  technician.  While  comparable  data  are  not 
available  to  represent  the  surface  navy,  data  from  industry  are  directly  relevant  and 
mirrors  that  of  the  naval  aviation  maintenance  workforce.  The  Labor  Department 
reports  that  approximately  140,000  mechanics  work  in  the  aviation  industry  today  and 
40,000  additional  mechanics  will  be  needed  by  2008.  In  order  to  fill  positions  necessary 
to  support  larger  fleets,  maintenance  facilities  are  being  forced  to  lower  hiring 
requirements  for  maintenance  personnel.  As  an  example,  American  Airlines  has 
recently  been  forced  to  lower  experience  requirements  from  4  to  2  years  (Chaparro, 
2001). 


In  a  similar  trend,  the  Navy's  Aviation  Depots  have  recently  consolidated  and, 
over  the  past  decade,  considerably  reduced  the  number  of  new  hires.  As  a 
consequence,  there  is  a  wealth  of  experience  that  is  now  ready  to  retire.  As  they  retire, 
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these  senior  technicians  take  with  them  that  corporate  knowledge  of  their  jobs  that  is 
not  represented  in  their  technical  publications. 

While  not  the  result  of  an  aging  workforce,  the  surface  navy  has  similar  issues 
with  the  skill  level  of  maintenance  technicians.  Trends  in  new  ship  design  are 
capitalizing  on  increased  automation  in  order  to  minimize  shipboard  manning. 

Manpower  is  the  single  greatest  lifecycle  cost  for  each  ship.  Thus,  programs  such  as 
DD(X)  and  CVN(X)  are  making  great  efforts  to  decrease  the  number  of  personnel 
onboard.  While  not  as  dramatic,  the  LPD  17  San  Antonio  Class  -  the  Navy's  next 
generation  amphibious  warfare  platform  -  has  also  made  considerable  efforts  to 
optimally  man  each  ship.  The  consequences  of  these  trends  equate  to  the  need  to 
empower  technicians  to  generalize  their  skills  beyond  any  one  system.  The  ability  to 
provide  this  empowerment  through  the  use  of  a  technician's  technical  manual,  a  major 
goal  of  the  current  effort,  requires  the  addition  of  performance  supporting  mechanisms, 
learning  opportunities  and  a  highly  intuitive  user  interface. 

Summary 

The  move  of  technical  manuals  to  the  electronic  environment  has  been  driven  by 
practical,  logistical  requirements.  Now,  however,  there  is  a  need  to  thoroughly  examine 
the  role  of  electronic  documentation  for  the  end  user.  It  is  remarkable  that  although 
the  technologies  embodied  within  IETMs  are  prevalent  in  other  common,  often-used 
systems  (e.g.,  Internet),  there  is  a  dearth  of  research  literature  related  to  the  use  of 
these  tools.  Where  currently  electronic  documentation  may  hinder  the  work  of 
maintenance  personnel,  future  products  must  exploit  the  capabilities  of  computers  to 
aid  technicians.  Further,  it  is  obvious  that  different  contexts,  domains,  and  tasks 
require  different  delivery  methods  and  varying  displays  of  information.  It  is  critical  for 
the  research  community  to  specify  those  relationships  and  provide  the  acquisition 
community  with  the  knowledge  to  purchase  the  right  system  for  the  right  job. 

ONR  began  highlighting  issues  related  to  use  of  IETMs  under  the  Condition- 
Based  Maintenance  (CBM)  program.  Three  ONR  sponsored  workshops  conducted 
between  1998-1999  addressed  the  topics  of  Machinery  Prognostics,  Personalized 
Maintainer,  and  Generalized  Maintainer.  In  particular,  the  Personalized  and  Generalized 
Maintainer  Workshops  specified  Science  and  Technology  (S&T)  issues  and  opportunities 
and  identified  potential  development  paths  for  enhancing  the  technology  and 
encouraging  its  use.  While  more  focused  on  the  interaction  of  IETMs  with  CBM 
technology,  there  were  a  number  of  issues  documented  that  are  directly  related  to 
current  efforts.  Of  the  five  themes  addressed  at  the  Generalized  Maintainer  Workshop, 
two  are  directly  related  to  current  research  activities:  1)  integrating  training  and 
performance  aiding  technologies  and  2)  expanding  and  refining  enabling  technologies. 
These  issues  will  be  addressed  in  subsequent  sections  of  this  report  in  the  context  of 
the  current  research  program. 
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INTELLIGENT  PERFORMANCE  SUPPORT  AND  TRAINING 

To  focus  research  on  technical  manual  issues  such  as  those  related  above,  a 
program  of  research  was  proposed  under  ONR's  Capable  Manpower  Future  Naval 
Capabilities  (FNC)  Program.  This  effort,  Intelligent  Performance  Support  and  Training, 
is  an  Enabling  Capability  of  the  Objective-Based  On-the-Job  Training  Supporting 
Capability.  The  product  associated  with  this  effort  is  an  authoring  capability  that  will 
allow  developers  to  build  IETMs  that  integrate  three  technologies  that  will  be  further 
developed  and  evaluated  under  this  program.  Specific  research  areas  include: 

*  the  use  of  device  models  and  intelligent  tutors; 

*  the  application  of  Latent  Semantic  Analysis  ( LSA )  for  search  and  navigation 
within  an  IETM;  and 

*  spoken,  natural  language  processing  and  wearable  computers  technologies  to 
support  hands-free  use. 

The  Research  and  Development  Issues  Associated  with  Interactive  Electronic 
Technical  Manuals  Workshop,  held  at  NAVAIR  Orlando,  Training  Systems  Division 
January  23  2002,  served  as  a  kick-off  meeting  for  the  Intelligent  Performance  Support 
and  Training  program.  The  goal  of  the  workshop  was  to  present  current  and  planned 
research  that  addressed  topics  related  to  IETMs.  Members  of  the  academic  community 
and  industry  as  well  as  fleet  representatives  provided  briefings.  The  workshop  agenda 
is  shown  in  Table  2.  A  list  of  workshop  participants  is  provided  in  Appendix  A. 

Three  of  the  presentations  provided  at  the  workshop  are  associated  with  the 
current  FNC  research  effort.  Those  presentations  included:  "IETM  Development  and 
Intelligent  Diagnostics,"  Dr.  Doug  Towne;  "Advance  Search  and  Display:  Dynamic 
SuperManuak  with  Latent  Semantic  Analysis,"  Dr.  Tom  Landauer;  and  "Spoken 
Language  Dialog  and  Improved,  Adaptive  Design  of  Advanced  IETM  Interfaces,"  Dr. 
Jane  Siegel  and  Dr.  Alex  Rudnicky.  Thus,  the  following  sections  are  provided  as  an 
overview  of  the  background,  goals,  and  products  of  each  of  these  efforts. 
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Presentation 

Overview 

lETMs  and  Maintenance  Safety  Issues 


IETM  Development  and  Intelligent 
Diagnostics/  Demonstration 


Advance  Search  and  Display:  Dynamic 
SuperManuals  with  Latent  Semantic 
Analysis 

Spoken  Language  Dialog  and  Improved, 
Adaptive  Design  of  Advanced  IETM 
Interfaces 

Royal  Navy  Advanced  IETM  Pilot  Study 


The  Use  of  lETMs  for  Operations, 
Doctrine,  and  Training 


Speaker,  Organization 

Dr.  Katie  Ricci,  NAVAIR  Orlando 

CDR  John  Schmidt,  Naval  Safety 
Center 

Dr.  Douglas  Towne,  University  of 
Southern  California/Behavioral 
Technology  Laboratory 

Dr.  Thomas  Landauer,  Knowledge 
Analysis  Technologies 


Dr.  Alex  Rudnicky  and  Dr.  Jane 
Siegel,  Carnegie  Mellon  University 


Mr.  Mike  Kendall,  Task-Based 
Technologies,  Inc. 

TMCM  (EOD)  Mark  Nelson, 
COMEODGRU  TWO 


Table  2:  Workshop  Agenda,  Speakers,  and  Organizations 
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DEVICE  MODELS  AND  INTELLIGENT  TUTORING 

For  any  given  system,  a  technical  manual  provides  the  information  needed  to 
support  a  technician  during  planned  or  corrective  maintenance  activities.  As  the 
documented  source  of  information  for  a  system,  the  technical  manual  is  very  often  the 
source  of  content  for  training  applications.  Thus,  technical  manuals  and  training  are 
closely  associated. 

The  training  pipelines  for  Navy  technicians  can  be  fairly  generic.  For  example,  a 
broad  technical  area  is  presented  (i.e.,  propulsion)  but  due  to  the  wide  range  of 
systems  in  use  in  the  navy,  a  specific  propulsion  system  may  not  be  taught.  It  is  not 
unusual  for  a  technician  to  arrive  on  station  and  be  assigned  to  a  system  that  is 
unfamiliar.  Further,  current  reduced  manning  trends  may  require  technicians  to 
perform  maintenance  on  a  wider  range  of  equipment  than  ever  before.  It  is  attractive, 
therefore,  to  embed  training  within  a  technical  manual  in  order  to  alleviate  gaps  in 
training.  In  fact,  current  IETM  applications  may  have  links  to  training  materials  to 
support  the  technician.  This  concept  is  even  specifically  described  in  the  Display  of  a 
Class  V  IETM:  "...simultaneous  access  to  many  differing  functions  (e.g.,  supply, 
training,  troubleshooting)"  (See  Table  1). 

The  notion  of  embedding  training  -  in  a  traditional  sense  -  within  a  technical 
manual  is  less  than  perfect.  Time  constraints  usually  prohibit  technicians  from  stopping 
procedures  until  a  training  application  is  completed.  Second,  the  building  of  expertise 
requires  practice  within  the  context  of  the  actual  system  -  a  feature  few  electronic 
training  applications  will  support.  Attendee  feedback  from  a  previous  Workshop  held 
January  2000  at  NAVAIR  Orlando,  overwhelmingly  expressed  the  notion  that  Interactive 
Courseware  (ICW)  linked  to  an  IETM  would  not  be  sufficient  or  useful  for  the  technician 
when  performing  maintenance  functions.  However,  a  more  comprehensive  solution  can 
be  obtained  when  technicians  are  provided  performance  support  as  well  as  training 
opportunities.  This  concept,  briefed  at  the  2002  workshop  by  Dr.  Doug  Towne,  is 
embodied  within  the  research  efforts  of  the  University  of  Southern  California/Behavioral 
Technology  Laboratory  (USC/BTL)  and  is  described  below. 


INTELLIGENT  TUTORING  FOR  FAULT  DIAGNOSIS 

Intelligent  tutors  for  fault  diagnosis  can  be  generalized  as  either  knowledge 
engineering  approaches  or  model-based  approaches  (Towne,  2002).  Knowledge 
engineering  approaches  rely  on  subject  matter  expertise  to  build  system  specific 
knowledge  for  fault  diagnosis.  A  notable  example  of  a  knowledge  engineering  approach 
to  tutoring  fault  diagnosis  is  Sherlock  (Lesgold,  Eggan,  Katz,  and  Rao,  2002;  Lajoie  and 
Lesgold,  1992).  Although  knowledge  engineering  approaches  provide  excellent  tutors, 
they  do  so  at  a  cost.  The  time  and  effort  needed  to  elicit  data  and  build  the  expert 
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system  is  quite  intensive  and  can  be  particularly  costly  when  addressing  complex 
systems. 

Unlike  knowledge  engineering  approaches,  model-based  approaches  represent 
the  target  system  via  a  computer  model  that  provides  domain-specific  functional 
information.  This  model  is  used  in  conjunction  with  a  domain-free  diagnostic  reasoning 
capability.  With  this  approach,  the  device  itself  is  separate  from  the  diagnostic 
capability.  Thus,  any  system  model  that  has  the  appropriate  data  to  share  with  a 
diagnostic  reasoning  capability  can  benefit.  In  the  current  research,  the  intelligent 
tutoring  and  performance  aiding  functions  make  use  of  a  model-based  approach. 

PROVIDING  MODEL-BASED  INTELLIGENT  TUTORING  WITHIN  AN  IETM 

Over  the  past  decade,  ONR  and  other  government  agencies  have  provided 
considerable  support  in  the  development  of  interactive,  simulation-based  systems.  The 
foundation  of  this  work  is  the  l//l//£>S software  (Munro,  1994;  Towne,  1994).  VIVIDS  \s 
a  general-purpose  simulation  authoring  system.  l//l//£>Sapplications  are  developed  and 
delivered  in  a  LINUX/UNIX  operating  system.  In  addition  to  VIVIDS,  USC/BTL  has 
developed  a  number  of  additional  applications  including  two  that  are  being  applied  to 
the  current  research  program:  ReAct  and  DIAG.  Much  like  VIVIDS,  ReAct is  a  general- 
purpose  simulation  authoring  and  presentation  system.  It  produces  a  device  model  and 
maintains  the  graphical  representation  as  the  model  changes  state.  Unlike  VIVIDS, 
ReAct  applications  are  produced  employing  Click  2  Learn's  ToolBook  in  a  Windows 
environment.  Diagnostic  Instruction  and  Guidance  ( DIAG)  inspects  the  device  model 
and  generates  intelligent  diagnostic  decisions  and  assessments. 

In  order  to  integrate  the  device  models  and  intelligent  tutors  into  an  IETM,  a 
control  shell  was  developed.  This  shell,  termed  the  Personal  Knowledge  System  ( PKS ), 
allows  the  presentation  of  ReAct  and  DIAG,  as  well  other  media  (e.g.,  Flash  animation, 
speech  technology,  etc.).  As  shown  In  Figure  1,  PKS  is  the  primary  interface  between 
training  or  aiding  media  and  the  end  user  (Towne,  2001). 
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Multimedia 

resources 


Speech 

technology 


Figure  1:  System  Architecture  (Towne,  2001) 


With  PKS  and  the  incorporation  of  DIAG  system  models,  schematics  and 
diagrams  within  the  technical  manual  can  reflect  the  general  system  architecture  as  well 
as  specific  configurations  under  any  condition  such  as  the  result  of  simulated 
malfunctions  or  user  actions.  This  is  in  stark  contrast  to  current  applications  using 
static  diagrams  that  require  the  end  user  to  mentally  simulate  component  behaviors  in 
order  to  determine  the  state  of  system  elements  and  the  connectivity  under  various 
conditions.  Thus,  conventional  diagrams  must  support  all  learning  or  performance 
situations  -  they  give  no  indication  of  signal  flows  or  pressure  status  as  the 
representation  does  not  correspond  to  any  real  condition  (see  Figure  2). 

Current  ideas  in  terms  of  overcoming  the  short  falls  of  displaying  large  diagrams 
on  a  computer  display  generally  deal  with  manipulations  to  the  viewing  field.  Such 
manipulations  include  scrolling,  zooming,  or  fisheye  functions  (Wang  &  Liang,  2000). 
These  methods  provide  better  resolution  and  incremental  viewing  to  a  diagram,  but  still 
maintain  a  static  display. 
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Figure  2.  Conventional  Diagram 


With  DIAG,  diagrams  change  to  support  specified  conditions  (see  Figure  3). 

For  any  given  system  mode,  the  diagram  can  explicitly  portray  system  elements  in  the 
status  that  applies  to  current  conditions.  Thus,  the  use  of  system  models  within  the 
IETM  can  dramatically  reduce  the  cognitive  workload  a  technician  traditionally  has  when 
attempting  to  mentally  represent  the  impact  of  a  change  in  system  status. 

DIAG  provides  a  number  of  functions  that  are  not  available  when  conventional 
diagrams  are  used.  These  functions  provide  the  user  with  both  performance  support 
aids  and  learning  tools.  During  FY03,  these  functions  will  be  evaluated  with  fleet  users. 
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Figure  3.  DIAG  Diagram 
Corrective  Maintenance 


When  undertaking  corrective  maintenance  tasks,  DIAG\s  able  to  advise  the  field 
technician.  Through  interaction  with  the  technician,  DIAG  will  analyze  initial 
observations  that  the  technician  has  made.  From  that,  DIAG  will  select  a  test  to 
perform  and  request  relevant  information  from  the  technician.  After  performing  the 
test,  the  technician  will  report  results  back  to  DIAG.  This  process  continues  until  DIAG 
recommends  a  repair  action.  When  DIAG  cannot  isolate  a  single  fault,  it  will  report  the 
fault  area  that  it  has  localized  (Towne,  2001).  The  projected  benefits  of  this  function 
include  the  potential  to  reduce  time  to  isolate  faults.  An  increased  accuracy  in  locating 
actual  faults  can  also  lead  to  a  reduction  in  the  consumption  of  spare  parts  as  greater 
accuracy  leads  to  less  guessing  on  the  part  of  the  technician  and,  therefore,  less  chance 
that  good  parts  will  be  replaced. 

DIAG  diagrams  reflect  system  connectivity.  During  corrective  maintenance, 

DIAG  provides  an  opportunity  for  the  technician  to  learn  how  the  system  functions. 
Diagrams  can  be  modified  as  the  technician  operates  upon  it  or  indicates  a  wish  to  see 
the  diagram  in  a  particular  mode.  The  projected  benefits  of  this  function  include  a 
reduction  in  time  needed  to  trace  inputs/outputs  and  increased  accuracy  in  tracing 
inputs/outputs. 
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DIAG also  functions  to  support  hyperlinks  from  schematic  elements.  Thus,  when 
a  technician  wishes  to  locate  technical  information  or  learn  the  physical  location  of  a 
schematically  depicted  part,  hyperlinks  are  located  on  the  diagrams  to  link  the  user  to 
that  information.  Because  the  technician  does  not  have  to  perform  search  operations 
within  the  document,  this  function  is  projected  to  reduce  the  time  needed  to  access 
needed  technical  information. 

Learning 

Because  DIAG  can  respond  to  user  actions,  it  can  allow  the  user  to  learn  how  the 
system  functions  in  a  particular  mode  and  practice  operating  the  target  system  in 
various  modes.  Thus,  it  supports  both  exploratory  learning  as  well  as  guided  instruction 
in  operation  and  maintenance.  Further,  the  DIAG  system  as  well  learns  from  diagnostic 
processes.  All  events  are  recorded  in  detail  allowing  DIAG  to  extract  and  incorporate 
new  symptom  information  into  its  knowledge  base. 

Projected  gains  from  these  capabilities  include  a  reduced  time  to  understand  how 
the  system  functions  and  to  attain  proficiency  in  operation.  Further,  repeated  exposure 
to  various  diagnostic  events  through  DIAG  allows  the  technical  manual  to  become  a 
storehouse  for  domain-specific  failure  data  (Towne,  2001). 
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LATENT  SEMANTIC  ANALYSIS  {LSA)  AND  SUPERMANUAL 

A  potential  --  but  largely  unrealized  -  tool  for  the  user  of  electronic  documentation 
provides  the  ability  to  search  for,  and  access  all  of  the  information  within  a  document 
that  is  relevant  to  the  user's  immediate  needs.  The  reason  why  this  promise  remains 
largely  unfulfilled  is  that  much  "electronic"  technical  material  is  not  organized  to 
facilitate  task-oriented  searches.  Today  the  bulk  of  electronic  documentation  consists  of 
paper  documents  that  have  been  scanned  into  a  PDF  format.  This  format  provides 
virtually  no  search  capability.  Recently  created  electronic  documents  authored  in  one  of 
the  existing  word  processor  formats  provide  an  automated  word  or  term  search 
capability  that  identifies  every  instance  that  the  word  or  term  is  used  in  the  document. 

A  subset  of  these  materials  also  include  hyper-linked  indexes,  tables  of  content,  and 
"hot"  words  and  phrases  that  take  the  user  to  a  glossary  of  definitions  or  other  helps. 
Finally,  some  exemplary  documents,  in  addition  to  the  features  described  above,  have 
been  manually  indexed  to  organize  the  material  by  level  of  detail,  or  level  of  relative 
difficulty  to  accommodate  the  specific  needs  of  experts  and  novices. 

All  of  these  search  schemes  have  a  significant  drawback  that  limits  their  usefulness. 
Simple  word  searches,  and  even  manually  indexed  documents  require  the  user  to  know 
(or  guess)  the  specific  words  used  by  the  author  to  represent  a  given  idea.  Thus,  a 
search  using  a  related  word  or  term  that  is  not  actually  used  in  the  text  in  that  form  - 
however,  relevant  to  the  topic  --  would  produce  no  results.  Latent  Semantic  Analysis 
(LSA)  provides  a  method  for  analyzing  and  successfully  searching  text  material  without 
requiring  the  searcher  to  decipher  the  author's  particular  terminology. 

WHAT  IS  LSA ? 

Latent  Semantic  Analysis  (LSA)  can  be  described  as:  (1)  A  method  for  obtaining 
estimates  of  the  substitutability  of  words  in  larger  text  segments,  and  of  the  meaning 
similarities  among  words  and  text  segments,  or  (2)  A  model  of  processes  and 
representations  that  underlie  the  acquisition  and  use  of  knowledge  (Landauer,  Foltz,  & 
Laham,  1998). 

LSA  is,  first,  a  statistical  technique  for  describing  and  comparing  the  similarity  of 
bodies  of  text.  It  does  this  by  applying  an  automatic  technique  for  extracting  and 
inferring  relations  of  expected  contextual  usage  of  words  in  passages  of  discourse.  It 
does  not  require  manually  constructed  dictionaries  or  ontologies.  Rather,  it  uses  only 
raw  text  that  is  parsed  into  words,  and  separated  into  meaningful  passages  such  as 
sentences  or  paragraphs. 

The  idea  underlying  LSA  is  that  the  similarity  of  the  contexts  in  which  a  word 
appears  is  a  reliable  indicator  of  the  similarity  of  the  meaning  of  words  to  each  other. 

By  processing  a  large  sample  of  language,  and  specifying  the  contexts  in  which  words 
occur,  LSA  can  organize  any  set  of  these  words  in  a  high  dimensional  semantic  space. 
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How  does  LSA  work? 


LSA  is  based  on  a  mathematical  technique  closely  akin  to  factor  analysis.  First, 
LSA  uses  the  frequency  with  which  two  words  are  used  within  a  portion  of  a  text  to 
establish  a  probabilistic  measure  of  semantic  association.  In  the  first  step  the  text  is 
represented  as  a  two-dimensional  matrix  in  which  the  rows  stand  for  unique  words  and 
each  column  stands  for  another  word,  a  sentence,  a  paragraph,  or  some  other  context. 
Each  cell  contains  the  frequency  with  which  the  word  of  its  row  appears  in  the  passage 
denoted  by  its  column.  Thus,  the  matrix  has  the  form  of  a  large  rectangular  table  with  a 
large  number  of  rows  and  an  even  larger  number  of  columns.  The  cell  entries  are  then 
weighted  by  a  function  that  expresses  the  word's  importance  in  the  passage  and  the 
degree  to  which  the  word  type  carries  information  in  the  domain  (Lee,  In  press). 

Next,  LSA  applies  a  statistical  technique  called  Singular  Value  Decomposition  (SVD) 
that  compresses  the  co-occurrence  information  into  a  smaller  space.  In  SVD,  a 
rectangular  matrix  is  decomposed  into  the  product  of  three  other  matrices.  The  first 
derived  matrix  describes  the  original  row  entities  as  vectors  of  derived  orthogonal  factor 
values.  The  second  derived  matrix  describes  the  original  column  entities  in  the  same 
way.  The  third  matrix  is  a  diagonal  matrix  containing  scaling  values  such  that,  when 
the  three  component  matrices  are  multiplied,  the  original  matrix  is  reconstructed.  The 
final  step  is  to  identify  relationships  that  underlie  the  pattern  of  occurrence  of  words 
across  passages.  In  theory,  semantically  related  words  and  passages  will  load  on 
similar  dimensions,  although  they  may  share  no  common  words  (Lee,  Submitted). 

How  has  LSA  been  used? 


LSA  was  originally  developed  for  the  task  of  information  retrieval,  but  has  been 
used  to  simulate  a  variety  of  language  tasks. 

-  It  has  taken  the  Test  of  English  as  a  Foreign  Language  and  performed  as  well  as 
non-native  English  speakers  who  were  successful  college  applicants. 

-  It  has  shown  an  ability  to  learn  words  at  a  rate  similar  to  humans.  It  has  even 
graded  papers  as  reliably  as  human  graders  (Landauer,  Foltz,  &  Laham,  1998). 

-  It  has  been  used  as  a  mechanism  for  evaluating  the  quality  of  student  responses 
in  an  intelligent  tutoring  system  and  its  performance  equals  that  of  human  raters 
with  intermediate  domain  knowledge. 

LSA  approximates  human  judgments  of  overall  meaning  similarity.  Based  on  a 
corpus  of  68,000  words,  LSA  acquired  a  working  vocabulary  at  the  same  rate  as  school 
children  (Landauer  &  Dumais,  1997).  LSA  also  provides  a  tool  for  extracting  the 
essential  semantic  components  of  information. 

LSA  has  been  used  to  interpret  and  score  a  student's  natural  language  input  in  an 
intelligent  tutoring  system  (ITS).  The  adequacy  of  Z.5/4S  reflection  of  human  knowledge 
has  been  established.  For  example, 
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-  its  scores  overlap  those  of  humans  on  standard  vocabulary  and  subject  matter  tests, 

-  it  mimics  human  word  sorting  and  category  judgments, 

-  it  simulates  word-word  and  passage-word  lexical  priming  data,  and 

-  it  accurately  estimates  passage  coherence,  learnability  of  passages,  and  the  quality 
and  quantity  of  knowledge  contained  in  an  essay  (Wiemer-Hastings,  2000). 

Additional  Research 


Landauer  and  Dumais  (1996,  1997)  have  proposed  that  LSA  constitutes  a 
fundamental  computational  theory  of  the  acquisition  and  representation  of  knowledge. 
As  a  statistical  method  for  characterizing  word  usage,  LSA  produces  measures  of  word- 
word,  word-passage,  and  passage-passage  relations  that  correlate  with  human  behavior 
involving  word  association  or  identifying  semantic  similarity.1  Landauer  and  Dumais 
maintain  that  LSAs  underlying  mechanism  can  account  for  the  ability  of  people  to 
acquire  more  knowledge  than  is  available  (Landauer  &  Dumais,  1996;  1997).  The 
principal  support  for  this  claim  has  come  from  using  LSA  to  derive  measures  of  the 
similarity  of  meaning  of  words  from  text.  Because  of  this,  and  other  capabilities  similar 
to  human  cognitive  performance,  Landauer  and  Dumais  have  proposed  LSA  be  used  to 
model,  test,  develop,  and  explore  fundamental  theoretical  implications  of  human 
information  processing. 

SUPERMANUAL 

SuperManual\s  based  on  the  empirically  validated  design  of  the  SuperBook 
reference-manual  browser  developed  at  Bellcore  in  the  early  90s  (Landauer,  1995). 
SuperBook  increased  speed  and  accuracy  on  information-dependent  tasks  by  large 
factors,  and  was  used  by  thousands  of  telecommunications  employees.  SuperBook's 
design  provided  a  number  of  innovative  techniques  for  displaying  technical  information 
such  as  the  dynamic  fisheye  table  of  contents  and  specialized  search  functions.  One 
critical  aspect  of  the  development  of  SuperBook  was  the  iterative  test  and  design 
methodology  (Landauer,  1995).  As  shown  in  Figure  4,  several  iterations  of  SuperBook 
were  required  before  finding  a  design  that  was  superior  to  paper-based  products.  This 
figure  illustrates  the  efficiency  of  finding  facts  in  a  technical  document  with  the  original 
printed  version  and  with  three  successive  versions  of  the  SuperBook  text  browser. 


1  This  correlation  may  be  the  result  of  the  word  choice  of  the  writers  of  the  text  corpus, 
which  reflects  the  way  people  typically  representation  meaning,  or  -  vice-versa  - 
peoples'  representations  of  meaning  may  reflect  the  statistics  of  word  patterns  they 
have  previously  read  and  heard. 
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Figure  4:  SuperBook Iterative  Design  (Landauer,  1995). 

Under  the  SuperBook  program,  a  number  of  features  were  researched.  In  some 
cases,  those  features  are  instantiated  in  current  applications.  For  example,  in 
SuperBook,  a  dynamic  "fisheye"  table  of  contents  is  created  from  the  heading  structure 
of  the  text.  The  table  of  contents  is  displayed  in  the  left  window  and  is  organized  by 
top-level  headings  that  can  be  opened  by  mouse  clicks  to  reveal  lower  level  sub¬ 
sections.  This  is  a  now  a  common  convention  in  electronic  documentation.  Other 
features  that  were  proven  to  be  sub-optimal  (e.g.,  Boolean  queries)  are  still  in  use 
today. 


The  objective  of  the  current  SuperManua! research  is  to  design  and  prototype 
better  ways  to  dynamically  organize,  present,  and  customize  information  for  particular 
tasks  and  individual  maintainer  knowledge  and  levels  of  expertise.  The  goal  is  to  make 
the  next  generation  of  IETMs  much  more  effective  as  job  performance  aids  by 
improving  the  maintainer's  ability  to  find,  use,  and  learn  from  their  task-relevant 
contents.  The  descriptions  below  summarize  some  of  the  current  research  issues  under 
an  ONR  funded  Phase  II  Small  Business  Innovative  Research  (SBIR)  award.  Further 
details  and  a  system  demonstration  can  be  found  at  http : //knowledge- 
technologies.  com/onr.html. 
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Meaning-Based.  Natural  Language  Search 

As  described  above,  LSA  is  a  key  component  to  the  SuperManual project.  LSA 
represents  a  major  innovation  to  the  original  SuperBook  desxqn.  LSAt  most  visible 
impact  in  SuperManualxs  the  meaning-based,  natural  language  search  capability. 
Search,  or  "find"  functions  in  current  electronic  documents  (e.g.,  Adobe)  target  each 
instance  of  that  exact  word(s)  in  a  linear  fashion  throughout  the  document.  Figure  5 
shows  a  word  search  carried  out  within  an  Adobe  document.  The  user  repeatedly 
presses  the  "next"  button  until  the  desired  area  of  the  document  or  content  is  found. 
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320-1.6.5.7  NLB  Type,  NLB  circuit  breakers  arc  similar  in  construction  to  ALB  breakers,  but  they  do  not  have 
any  overcumeni  trip  capability.  These  devices  are  used  to  break  load  by  operating  a  manual  toggle  switch  handle. 
They  are  designed  for  use  on  low  voltage  power  and  lighting  distribution  systems. 

320-1  6.6  FUSES.  A  Q^3|confiisfcs  of  a  metal  conductor  inserted  into  a  tube  of  glass  or  other  insulating  mate¬ 
rial,  that  melts  when  the  current  through  the  conductor  exceeds  the  rated  level,  opening  the  circuit.  Metal  ferrules 
at  each  end  of  the  fuse  make  contact  with  fuse  clips  or  contacts  in  the  carriage  of  a  screw -type  fuse  holder.  Fuses 
arc  used  as  protective  devices  in  power  and  lighting  circuits.  They  axe  in  some  user  equipment,  where  their  per¬ 
formance  is  preferred  over  that  of  a  circuit  breaker  or  their  use  is  considered  more  economical.  Motors  rated  up 
to  7-1/2  horsepower  (hp)  are  often  supplied  from  group  control  centers  having  30-ampere  fuses.  Fuses  are  not 
used  in  4 SOY  circuits  supplying  motors  in  excess  of  7-1Z2  hp.  Unlike  circuit  breakers,  a  fuse  must  be  replaced 
when  it  fails.  Sec  NSTM  Chapter  30®  for  directions  on  removing  and  replacing  fuses. 

320-1.6.6.1  Fuse  Characteristics.  There  are  three  types  of  fuses  with  one  of  the  following  characteristics: 

a  Characteristic  A  -  normal  bl owing 

b.  Characteristic  B  -  time  lag  (slow  blow) 

c.  Characteristic  C  very  high  (100.000  amperes)  interrupting  capacity 

C  fuses  arc  used  where  the  available  lau.lt  current  exceeds  the  10,000  amperes  maximum  interrupting  capac¬ 
ity  of  A  or.B  ffcsea.  i  ’ 


If  A  or  B  foica  are  used  in  an  application  requiring  lype  C  fuses,  the  fust 
may  explode,  taming  Gure  and  pcriunnd  injury  and  taquipmcnl  danuigd. 
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Figure  5.  Adobe  Word  Search 


Unlike  a  traditional  search,  SuperManual  {as  did  SuperBook)  shows  all 
occurrences  of  the  search  terms  posted  against  the  dynamic  table  of  contents.  Figure  6 
shows  the  same  search  within  a  SuperManuahevsxon  of  this  document.  Every  place 
that  the  query  word  appears  in  the  document  is  indicated  in  the  table  of  contents,  thus 
providing  a  context  for  the  user  to  select  the  appropriate  section  to  select.  Furthering 
this  in  SuperManualxs  the  addition  of  LSA  -  not  only  will  the  table  of  contents  show 
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where  that  exact  word  exists  in  the  document,  it  will  also  show  areas  where  meanings 
closely  associated  to  the  query  term  also  exist. 
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320-J  .6.16  FUS ES.  A  fiis«  consists  of  u  metal  conductor  tokened  into  a  tube  of  glass  at  oilier  insulating  material,  that  mets  when  the  cur 
eonduaof  exceeds  the  rated  level,  opening  the  ctreu.it.  Metal  ferrules  at  each  end  of  the  fuse  nuke  contact  with  fuse  clip*  or  contacts  in  d: 
sctew-lype  fuse  hoJdef.  Fuses  me  used  as  protective  devices  in  power  and  lighting  circuits  .  They  me  in  some  user  equipment,  -where  the 
preferred  rover  that  of  a  circuit  breaker  or  their  use  is  considered  more  economical.  Motors  rated  up  to  7-1/2  horsepower  (hp)  are  often  si 
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replaced  when  it  fads.  Sec  NSTM  Chapter  300  for  directions  on  removing  and  replacing  fuses 


320-1.6  6.1  Fuse  Chaact eristics  There  are  three  types  of  fuses  with  o: 
a.  Characteristic  A  normal  blowing 


Characteristic  B  -  time  lag  (slow  blow) 
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Characteristic  C  -  very  high  (100.000  amperes)  interrupting  capacity 

C  fusia  are  used  where  the  m  ailable  fault  current  exceeds  the  10.000  amperes  maximum  mtemiptmg 


ity  of  A  01B  fuses. 

WARNING 

If  A  or  B  fuses  aw  used  in  an  appikauon  requiring  type  C  fiists,  the  fuse  may  explode,  causing  fire  and  personnel  injury  and  equipiueni 

320-1 .6.7  CURRENT  TIME  SENSING  AND  SIGNALING  DEVICE.  A  current  time  sensing  (CTS) device  is  designed  for  use  with  J 
-A400,  and  -A  KUO  shunt  trip  eheuit  breakers.  They  n re  presently  installed  only  for  protecting  400-112 .systems  that  are  supplied  by  solid- 
changers  (SSFC)  because  tliesc  SSFCs  are  current  limited  and  self  protect  before  they  can  actuate  thermal  or  magnetic  trip  circuit  Iweakcj 
time-current  cliaraclerislics  are  more  precise  and  adjustable  than  those  of  AQB  circuit  breakers  with  thermal  and  magnetic  trips.  The  CTs 
the  breaker's  time-current  characteristics  witii  a  selectable  time  delay  for  fault  currents  below  the  breaker's  instantaneous  pickup.  A  CTS 
through  its  associated  circuit  breaker  and  directly  activates  die  shunt  trip  rtvedtanism  of  the  circuit  breaker  in  response  to  overload  and  fa 
with  AQB-A  1 02  ettpuh  breakers  (doctronic  uip)  do  not  need  GTS  monitors.  400-Hz  systems  supplied  by  motor  generator  sets  do  non; 
because  they  are  not  current  limited. 


Figure  6.  SuperManua!  Sea  rch 


Minimum  Path  to  Knowledge 

When  preparing  to  perform  a  specified  maintenance  task,  technicians  may  desire 
a  review  of  the  supporting  information  for  that  task.  This  review  might  be  significantly 
time  intensive  as  it  requires  skimming  through  an  entire  document  to  find  applicable 
sections.  Currently  under  investigation  in  the  SuperManua! program  is  an  algorithm, 
termed  "Optimin,"  designed  to  find  the  sequence  of  subsections  or  paragraphs  in  a 
manual  that  will  propose  a  path  from  either  its  beginning  or  from  a  user-selected 
starting  point  to  the  target  section.  The  suggested  sequence  of  steps  is  then  displayed 
in  the  dynamic  table  of  contents.  The  results  of  this  search  would  be  used  as  a  guide 
to  finding  the  needed  information  to  perform  a  given  task. 

Keyword  and  Sentence  Summaries 

The  LSA  generated  keyword/sentence  summary  will  provide  an  icon  at  the  end  of 
a  table  of  contents  title.  When  this  icon  is  clicked,  a  pop-up  box  will  display  either  a  3 
to  5  word  automatically  generated  keyword  summary  or  a  best-sentence  summary  of 
the  section.  The  "best-sentence"  is  generated  by  LSA  by  comparing  the  meaning  of 
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each  sentence  in  section  with  each  other  sentence  and  finding  the  one  whose  average 
similarity  to  all  others  is  greatest. 

Adaptive  Indexing 

When  using  a  traditional  search  convention,  such  as  a  find  feature,  each 
instantiation  of  the  term  or  terms  can  be  found.  However,  there  is  often  the  case  when 
the  information  needed  may  not  be  the  exact  term  used  in  the  search.  The  system 
keeps  track  of  the  words  that  users  try  that  do  not  succeed.  If  and  when  the  user 
finally  finds  a  sought  item,  the  system  asks  whether  some  of  the  user's  words  that  did 
not  find  the  desired  content,  but  should  have,  ought  to  be  added  to  the  list  of  index 
terms  for  that  item.  If  the  user  concurs,  the  new  key  words  are  promoted  to  definitive 
pointers  that  remain  available  for  that  and  other  users  in  the  future. 
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SPOKEN  LANGUAGE  INTERFACE  AND  WEARABLE  COMPUTING 


In  recent  years,  technology  has  progressed  to  enable  mobile  computing. 
Certainly,  platforms  such  as  the  personal  palm  computers  provide  access  to  data,  email, 
and  scheduling  applications.  However,  for  complex  operations  such  as  corrective 
maintenance  actions,  the  full  capability  of  mobile  computing  has  yet  to  be  exploited. 

The  requirements  of  maintenance  are  vastly  different  from  those  of  applications  on 
palm  computers.  To  say  the  least,  the  real  estate  available  on  hand-held  computers  is 
insufficient.  Further,  the  key  to  mobility  is  the  ability  to  perform  hands  free  of  the 
computer.  The  challenges  associated  with  developing  such  a  system  is  the  advanced 
capabilities,  such  as  spoken  language  interfaces  and  the  delivery  of  IETM  information 
on  a  wearable  computer  that  is  functional,  reliable,  and  safe. 

Currently,  two  inter-related  capabilities  are  being  investigated  through  the 
Intelligent  Performance  Support  and  Training  program  by  researchers  at  Carnegie 
Mellon  University  (CMU).  The  first,  spoken  language  interface,  relates  to  the  voice 
interaction  mode  by  which  technicians  extract  information  from  and  input  information  to 
the  technical  manual.  Closely  related  to  this  is  the  capability  that  allows  the  technician 
freedom  of  movement  without  losing  contact  with  technical  documentation.  Wearable 
computers  with  head-worn  displays  provide  this  mobility.  The  objectives  of  the  CMU 
research  are  to: 

•  assess  the  utility  of  spoken  language  dialog  and  adaptive  design  of  advanced 
IETM  interfaces; 

•  conduct  experiments  and  field  usability  evaluations  to  rapidly  and 
systematically  increase  knowledge  about  adaptive,  advanced  IETM  interfaces; 
and 

•  develop  initial  principles  of  multi-media  advanced  IETM  systems  in  terms  of 
interaction  options  to  support  training  and  performance  aiding  and  delivery 
on  mobile/wearable  computers. 

SPOKEN  LANGUAGE  INTERFACE 

The  Carnegie  Mellon  Speech  Group  has  a  long  history  of  developing  and 
researching  speech  systems.  In  1976,  the  Dragon/Harpy  system  investigated  the 
recognition  of  a  large  vocabulary.  Speaker-independent  continuous  speech  was 
researched  with  the  Sphinx  system  in  1987.  Spoken  language  information  retrieval  was 
achieved  with  the  A  TIS  system,  and  the  Speech  Wear  system  addressed  wearable 
speech  systems.  Most  recently,  the  DARPA  Communicator 'program  addresses  dialog- 
based  problem  solving.  Using  a  travel-planning  tool  accessible  through  phone  at  1-877- 
CMU-PLAN,  callers  can  plan  travel  through  the  assistance  of  Communicator.  Data  are 
collected  to  analyze  dialogs  as  they  relate  to  a  problem-solving  domain.  Research 
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issues  include  architectures  for  spoken  language  systems,  dialog  management,  and 
language  generation. 

The  current  research  is  working  with  the  Carnegie  Mellon  Speech  Group's 
Symphony  System .  The  goal  is  to  provide  a  natural  spoken  interface  -  not  a  simple 
mapping  between  spoken  words  and  graphical  user  interface  elements.  The  vision  of 
the  Symphony  program  is  to  allow  for  natural,  voice-driven  access  to  technical  manual 
materials.  To  support  the  technician,  the  system  must  provide  assistance  (other  than 
reading  the  technical  manual)  in  diagnostic  procedures.  That  is,  it  must  provide  a 
mentoring  capability  that  will  shadow  maintenance  activities,  offer  procedure  advice  and 
provide  historical  data. 


WEARABLE  COMPUTERS  AND  OPTIMIZED  DISPLAYS 

Prior  research  conducted  for  ONR  provides  valuable  input  to  inform  future  design 
efforts.  In  particular,  data  were  collected  at  China  Lake  Naval  Weapons  Station  from  8 
maintainers  of  the  F/A-18  aircraft  (Siegel,  Hyder,  Moffett,  &  Nawrocki,  2000).  Two 
IETM  interfaces  were  studied.  The  first  was  a  windows/web  browser  design  typical  of 
current  IETM  interfaces.  The  second  was  a  highly  graphical  interface.  Both  interfaces 
were  used  in  conjunction  with  a  dial  I/O  device  developed  at  CMU  and  a  head-worn 
display  attached  to  the  maintainer's  cranial  cap  (see  Figure  7). 


Figure  7.  Wearable  Computer  with  Voice  and  Dial  Interface 
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This  study  concentrated  on  three  specific  IETM  challenges:  navigation,  use  of 
magnification,  and  cautions.  The  graphical  interface  was  designed  to  enhance  these 
challenges.  Specifically,  this  interface  design  displayed  procedures  using  a  navigational 
frame  providing  high  level  steps  as  well  as  access  to  lower  level  details  that  might  be 
needed  by  a  novice.  The  display  completely  hid  all  other  operating  system  buttons 
(e.g.,  Windows  "Start"  button)  that  could  potentially  lead  to  confusion  or  frustration  if 
inadvertently  selected  by  novice  computer  users.  The  display  utilized  an  icon  that 
allowed  images  to  expand  the  width  of  the  screen,  providing  magnification  and  scrolling 
options  as  well.  Finally,  cautions  were  embedded  as  steps  and  displayed  as  icons.  As 
the  step  is  selected,  the  alert  appeared  -  however,  it  did  not  obscure  procedures,  as  is 
common  in  other  IETMs. 

While  it  is  not  possible  to  make  conclusions  given  a  small  sample  size,  the  data 
from  this  study  suggest  that  novices  preferred  the  web/browser  design  -  perhaps 
because  it  is  more  like  paper  documentation.  Experts,  however,  were  more  likely  to 
prefer  the  graphical  design  and  stated  that  this  design  gave  them  a  better  sense  of  the 
whole  task.  All  of  the  participants  agreed  that  current  alert  displays  are  less  than 
optimal  and  preferred  methods  of  displaying  alerts  without  obscuring  the  display. 
Similarly,  all  participants  indicated  a  need  for  magnification  and  scrolling  for  small 
images  and  large  schematics. 

While  the  IETMs  studied  received  positive  feedback,  comments  as  to  what  the 
maintainers  liked  least  included  the  dial  I/O  device  and  the  display.  There  were 
instances  noted  where  maintainers  would  turn  the  dial  too  quickly  and  jump  over 
information.  Further,  there  was  note  of  uncertainty  as  to  how  to  navigate  backwards  or 
forwards.  The  mobility  and  hands-free  work  afforded  by  the  wearable  were 
enthusiastically  welcomed.  However,  there  is  additional  research  necessary  to  find  the 
optimal  hardware  components.  A  more  recent  iteration  of  CMU's  wearable  computer, 
"SPOT",  is  presented  in  Figure  8.  Specifications  of  the  SPOT  system  are  provided  in 
Table  3. 
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SPOT  SPECIFICATIONS 


Processor 

206  MHz  SA-1110 

SA-111 

Memory 

256  MB  SDRAM 

Flash  RAM 

64  MB 

PCMCIA 

1  Type-II 

Compact  Flash 

1  Type-II 

Display 

DVI  1.0  Output 

Serial  I/O 

1  RS-232 

USB 

1  Type  A 

IEEE-1394 

2  Powered  Ports 

Channels 

Audio 

Stereo  In/Out 

Table  3.  SPOT  Specifications 
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WORKSHOP  CONCLUSIONS 

The  NAVAIR  Orlando,  Training  Systems  Division,  hosted  an  Office  of  Naval 
Research  (ONR)  sponsored  workshop  addressing  research  and  development  issues  for 
advanced  Interactive  Electronic  Technical  Manuals  (IETMs).  The  purpose  of  this 
workshop  was  to  provide  a  forum  for  academia,  government,  and  Fleet  personnel  to 
discuss  ongoing  and  planned  research  related  to  advanced  electronic  maintenance 
documentation  for  performance  support  and  training. 

The  workshop  highlighted  several  areas  that  currently  pose  difficulty  for 
developers  and  users  of  IETMs.  Of  those  issues,  several  are  being  addressed  in  the 
current  research  effort:  troubleshooting  aids;  search,  navigation  and  interaction  tools; 
and  mobility.  The  development  and  evaluation  of  these  products  are  progressing 
separately.  As  they  mature,  they  will  be  integrated  to  work  together.  As  initially 
planned,  that  collaboration  was  to  begin  in  FY04.  This  plan  did  raise  some  concern 
related  to  the  fact  that  a  myriad  of  proprietary  software  products  exists  today.  The 
notion  of  further  development  of  three  separate  products  without  effort  related  to 
integration  was  seen  as  increasing  the  likelihood  of  ending  up  with  three  separate 
systems.  Accordingly,  collaboration  efforts  were  moved  up  from  FY04  to  FY03. 

The  presentation  and  demonstration  given  by  Dr.  Towne  highlighted  the  use  of  a 
model-based  approach  to  intelligent  tutoring  within  an  IETM.  Clearly,  the  DIAG 
component  of  the  PKS  system  provides  enormous  potential  in  making  technical 
documentation  less  mentally  demanding  on  the  technician.  Flowever,  concerns  were 
expressed  that  the  development  of  a  DIAG  model  is  time-intensive  and  requires  the 
appropriate  knowledge  of  the  system  and  DIAG  software.  Evaluations  conducted 
during  FY03  will  hopefully  provide  the  needed  data  to  show  the  benefit  of  a  DIAGIEUA. 
As  the  use  of  computer-aided  design  (CAD)  systems  increase,  there  is  potential  to 
convert  system  information  directly  to  a  modeling  program  such  as  Re  Act 

Data  presented  from  the  Naval  Safety  Center  indicate  publications  are  a  factor  in 
25%  of  aviation  mishaps.  Further  work  to  define  the  nature  of  the  impact  of 
publications  on  aviation  mishaps  is  now  underway  under  the  support  of  this  research 
effort.  Additional  data  from  the  Naval  Safety  Center  show  that  the  updating  process  for 
aviation  technical  manuals  is  excruciatingly  slow.  Similar  data  exist  for  the  surface 
technical  manual  community.  Currently,  changes  to  technical  manuals  begin  with  a 
"Technical  Manual  Deficiency  Report,"  located  at  the  rear  of  each  manual.  Technicians 
complete  the  report  describing  the  problem  or  the  change  needed  and  submit  the 
report  to  the  "In-Service  Engineering  Agent  (ISEA)"  for  that  piece  of  equipment.  If  the 
problem  is  safety  related,  then  the  change  might  happen  rather  quickly.  If  the  change 
is  not  safety  related  and  the  equipment  is  legacy  (i.e.,  not  funded),  changes  build  up 
and  might  get  implemented  in  time  (i.e.,  every  5  years  or  so).  Flowever,  it  is  likely  that 
without  funding,  changes  will  not  be  validated  and  thus,  go  ignored. 
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Even  with  new  equipment,  technical  manual  changes  are  not  made  often,  as  the 
cost  of  engineering  verification  is  still  an  issue.  The  ISEA  would  probably  wait  until 
there  were  a  number  of  deficiency  reports  before  funding  an  engineer  to  verify  the 
changes.  The  verified  changes  are  currently  sent  out  and  are  recorded  on  a  'change 
page'  at  the  beginning  of  each  manual. 

While  the  updating  and  change  processes  that  a  manual  must  go  through  are 
not  the  focus  of  this  current  effort,  the  ability  of  maintainers  to  quickly  forward  potential 
erroneous  information  and  the  ability  of  the  technical  manual  authority  to  reply  with 
updated  information  must  be  available  within  any  IETM.  Further  attention  will  be  given 
to  this  area  as  progress  begins  on  the  development  of  an  authoring  tool. 

The  integration  of  device  models  within  technical  manuals  provides  potential 
benefits  in  alleviating  some  of  the  burden  of  technical  manual  updates.  In  fact,  initial 
deficiencies  could  be  reduced  dramatically.  One  issue  raised  recently  in  the 
development  of  the  PKSjDIAG  demonstration  for  the  S-3B  Nose  Wheel  System  was  the 
inaccuracy  of  the  original  technical  manual.  This  problem  became  evident  when  the 
model  generated  for  the  system  did  not  behave  in  accordance  with  expectations. 
Subsequent  consultations  with  subject  matter  experts  revealed  the  inaccuracy  in  the 
model  stemmed  from  statements  in  the  technical  manual  that  were  erroneous.  Once 
these  errors  were  corrected  in  the  model,  the  model  behaved  correctly.  Thus,  if  future 
manuals  had  the  benefit  of  device  models,  errors  in  the  documentation  could  be 
detected  prior  to  release. 

A  problem  clearly  evident  with  current  IETM  products  is  the  avoidance  shown  by 
technicians  to  use  IETM  products  that  are  perceived  as  complex  and  unintuitive.  For 
certain,  technology  has  accelerated  tremendously  and  has  provided  products  that 
remain  untested  for  effectiveness  and  efficiency.  The  lack  of  use  of  current  IETMs 
noted  at  the  workshop  underlies  the  importance  of  applying  new  technology  to  IETMs. 
Current  capabilities  utilize  computers  as  a  substitute  for  paper  rather  than 
capitalizing  on  computing  resources.  IETMs  must  allow  computers  to  do  functions  that 
computers  can  do  betterthm  paper.  There  are  large  gaps  that  the  research 
community  must  fill  to  truly  capitalize  on  the  promise  of  the  electronic  environment. 

The  benefits  of  reduced  space  needed  to  house  documentation  and  reduced  weight  for 
heavy  paper  manuals  will  be  quickly  eroded  when  copy  machines  and  associated  paper 
products  are  needed  throughout  ships'  spaces.  Further,  the  benefits  of  rapid  updates 
to  electronic  documents  are  negated  when  technicians  print  paper  versions  and  thus, 
ignore  subsequent  updates  to  the  electronic  version. 
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